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Abst  ract 

Two  theoretical  approaches  are  considered  lor  dealing  with  omitt.-' 
responses.  A  simple  modification  of  the  usual  item  response  theory  mode1, 
leads  to  internal  contradictions.  An  alternat  ive  model  is  sucpe.-  t  e-i 
some  special  cases  are  investigated. 


Maximum  Likelihood  Estimation  of  Item  Response  Parameters 


When  Some  Responses  Are  Omitted* 

In  item  response  theory,  the  frequency  distribution  of  the  responses 
of  a  single  examinee  to  n  dichotomous  items  is  commonly  given  as 

n  u .  1-u . 

f(ulS...,un)  =  n  Pi1Q.  1  (1) 

“  i=l 

where  =  1  -  P^  ,  P^  is  the  probability  of  a  correct  response  by 
the  examinee  to  item  i  ,  and  where  u.  =  0  or  1  denotes  his  score  on 

l 

the  item.  When  the  examinee  omits  (fails  to  respond  to)  an  item,  this 
formula  cannot  be  used.  The  purpose  of  this  article  is  to  explore  two 
theoretical  approaches  that  attempt  to  cope  with  omitted  responses. 

Section  1  presents  some  preliminary  considerations.  Section  2 
shows  that  a  conveniently  simple  application  of  (1)  leads  to  internal 
contradictions.  Section  3  considers  a  possible  rigorous  mathematical 
model.  Sections  A  and  5  show  that  this  model  yields  very  reasonable 
results  in  two  special  cases  that  are  sufficiently  simple  to  be  already 
familiar  to  us. 

Two  conclusions  are  reached. 

1)  A  simple  modification  of  the  usual  item  response  theory  model 
(1)  does  not  apply  when  the  examinee  has  the  option  of  omitting  or 
responding  at  random. 

*This  work  was  supported  in  part  by  contract  N00014-80-C-0402, 
project  designation  NR  150-453  between  the  Office  of  Naval  Research  and 
Educational  Testing  Service.  Reproduction  in  whole  or  in  part  is 
permitted  for  any  purpose  of  the  United  States  Government. 
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2)  A  model  containing  five  item  parameters  and  two  examinee 
parameters  seems  reasonable.  Even  if  some  of  these  parameters  should 
prove  difficult  to  estimate  accurately  in  practical  work,  the  model 
should  be  useful  for  clear  thinking  about  omitted  responses. 

1.  Preliminary  Considerations 

If  test  score  is  the  number  of  right  answers,  an  examinee  who 
omits  responses  on  a  multiple-choice  test  necessarily  lowers  his 
expected  test  score.  Mathematical  modeling  of  such  (usually  irrational) 
behavior  will  not  be  attempted  here. 

We  will  deal  instead  with  the  case  where  the  'formula  score' 
x  -  W/ (A  -  1)  is  to  be  used,  x  being  the  number  of  right  answers, 

W  the  number  of  wrong  answers,  and  A  the  number  of  choices  per  item. 
In  this  case,  the  examinee  who  wishes  to  maximize  his  expected  score 
should  not  omit  any  items  on  which  his  chance  of  success  is  greater  than 
1/A  .  If  he  can  do  no  better  than  a  random  guess  on  an  item,  his 
chance  of  success  equals  1/A  and  his  expected  test  score  will  be 
the  same  whether  he  omits  the  item  or  guesses  at  random.  He  may  omit 
or  guess  at  random,  as  he  pleases.  If  an  examinee  is  forced  to 
respond  to  an  item  that  he  has  omitted,  his  chance  of  success  is 
assumed  to  be  1/A  . 

Note  that  in  (1)  represents  what  the  statistician  knows 

about  item  i  before  it  is  administered  to  a  given  examinee.  After 
the  examinee  has  read  the  item,  he  may  know  the  correct  answer  with 
more  or  less  certainty,  he  may  be  misinformed  on  the  item  and  thus 
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answer  incorrectly,  or  he  may  guess  at  random  with  chance  of  success 
1/A  .  One  should  not  conclude  in  such  cases  that  P  is  1,  or  that  it 
is  0,  or  that  it  is  1/A  . 


The  Case  of  Equivalent  Items 


Items  are  called  equivalent  when  they  have  identical  P.  5  P  ^  (  r- )  = 
Prob(u^  =  1 j  0 )  =  P  at  all  ability  levels  9  .  We  will  consider  the 
case  where  both  the  mathematical  form  of  P.  and  the  numerical  values 

l 

of  the  (item)  parameters  defining  P  are  known. 

When  P^  in  (1)  is  replaced  by  P  ,  the  frequency  distribution 
of  the  u.  becomes 

l 


lu .  n-"u. 

f (u1  ,  .  . .  ,u  )  =  P  1Q 
In 


(2) 


n 

Since  x  -  I  u.  is  the  number-right  score,  we  see  that  when  x  is 
i=l  1 

given,  the  distribution  of  in  (2)  does  not  depend  on  . 

Thus  (as  is  well  known)  when  all  items  are  equivalent,  x  is  a 
sufficient  statistic  for  estimating  9  .  Any  inference  about 
should  depend  only  on  x  The  frequency  distribution  of  x  is  the 
familiar  binomial 


f (x)  =  (  n  )  PXQn"x  (  x  =  0,1,..., n  ) 

X 


(3) 


-4- 


Suppose  now  chat  a  given  examinee  will  either  omit  a  certain  item 
or  else  choose  his  response  at  random.  According  to  the  model,  if  he 
chooses  his  response  at  random,  his  number-right  score  x  will  be 
distributed  as  in  (2). 

Suppose,  on  the  other  hand,  he  decides  to  omit  this  one  item  (and 
r  others).  Denote  his  number-right  score  on  the  remaining  n  -  1 
items  by  k  (  k  =  0, . . . , n  -  1  ) .  Let  g(k)  denote  the  f  requency 
distribution  of  k  for  this  examinee.  We  hope  to  find  the  mathematical 
form  of  g(k)  so  that  we  can  use  it  to  get  a  maximum  likelihood 
estimator  of  0  . 

Since  his  probability  of  success  on  the  omitted  items  is  C  E  1/A  , 
there  is  a  mathematical  relation  between  f(  )  and  g(  )  : 


f ( 0)  =  (1  -  C)g(0)  , 


\ 


f(k)  =  Cg (k  -  1)  +  (1  -  C)g(k)  (  k  =  1,2 . n  -  1  )  ,  ^  (4) 

f (n)  =  Cg (n  -  1) 


If  the  f(  )  are  considered  as  known,  the  g(  )  as  unknowns  to  be 
determined,  we  have  here  n  +  1  linear  equations  in  n  unknowns. 
Written  in  matrix  form,  (4)  becomes 
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It  seems  reasonable  that  g(k)  determined  from  (4)  or  (5)  provides 
the  likelihood  function  from  which  the  examinee's  0  may  be  estimated 
by  maximum  likelihood. 

A  standard  investigation  (for  example,  Aitken,  1944,  Section  30) 
shows  that  these  n  +  1  linear  equations  have  no  solution  (are 
mutually  inconsistent)  unless  P  =  C  .  Thus,  excluding  the  uninter¬ 
esting  case  where  P  =  C  ,  when  f(  )  is  given  by  (3)  no  possible 
frequency  distribution  g(k)  (for  the  examinee's  responses  to  non¬ 
omitt  "■d  items)  can  exist. 

Model  (1)  can  be  applied  by  replacing  omitted  responses  by  random 
responses.  This  introduces  additional  error  into  the  data,  however. 

If  omits  are  not  replaced  by  random  responses,  the  usual  item  response 
function  model  (1)  cannot  fit  the  data.  We  have  only  proved  this  for 
tests  composed  of  equivalent  items,  but  the  conclusion  is  presumably 
general,  especially  since  any  test  may  contain  a  few  items  that  are 
statistically  equivalent. 
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3.  A  Model  for  Response  or  Omission 

The  discussion  in  Section  1  suggests  the  following  model.  Let 

Ri ('.'3)  denote  the  probability  that  examinee  a  ,  at  ability  level 

,  feels  no  preference  for  any  of  the  available  responses  to  item 
a 

i  :  if  he  responds  to  item  i  ,  he  will  respond  effectively  at 

random.  For  brevitv,  we  will  sav  that  R.("‘  )  is  the  probability  of 

1  a 

(total)  ignorance  on  item  i  .  It  may  be  thought  of  as  the  proportion 

of  examinees  at  ability  level  ••  who  are  ignorant  on  item  i  . 

Let  P .  ( ■'  )  denote  the  conditional  probability  that  examinee 
x  a 

a  will  respond  correctly  to  item  i  given  that  he  Js  not  ignorant. 
Thus 

P . ( ?  )  =  Prob(u.  =  linot  ignorant) 

1  a  ia  b 

If  the  examinee  omits  item  i  ,  we  will  denote  this  bv  w .  =  1  ; 

ia 

if  he  responds  to  item  i  ,  w.  =  0  .  Let  u  denote  the  probability 

ia  a 

that  examinee  a  will  omit  an  item  of  which  he  is  ignorant: 

a!  =  Prob(w.  =  l|  ignorance) 
a  ia 

Since  we  are  concerned  with  a  single  examinee,  we  will  drop  the 
subscript  a  .  As  before,  let  C  5  1/A  denote  the  probability  of 
success  by  random  guessing: 

C  =  ProbCu^  =  1 i w .  =  0,  ignorance) 

Let  a  bar  above  a  symbol  denote  its  complement,  for  example, 


P  =  1  -  P,  . 
1  i 


There  are  three  exhaustive  and  mutually  exclusive  events,  denoted 
by  (  w.  =  1  ) ,  (  =  0  ,  ut  =  1  ),  and  (  w^  =  0  ,  u  =  0  ).  The 

unconditional  probability  of  omitting  is  clearly 

Prob (w^  =  1)  =  -R.  .  (6) 

A  correct  answer  occurs  with  probability  ~C  when  the  examinee  is 
ignorant  and  with  probability  when  he  is  not  ignorant,  so 

Prob (u .  =  l,w.  =  0)  =  R.Tc  +  R.P.  .  (7) 

li  ill  v  ’ 

Similarly, 

Prob(u  =  0,  w .  =  0)  =  R.“c  +  R.P.  .  (S') 

li  ill 

The  right  sides  of  (6),  (7),  and  (8)  sum  to  1,  as  they  should. 

The  joint  distribution  of  w^  and  u.  (  i  =  l,2,...,n  )  may 
be  written 

n  w  _  _  u.w._  u.w. 

L  =  H  (R .  w)  (P.R.  +  C.R.)  1  1  (P  ,R .  +  C.ZR.)  1  1  (91 

^_^i  ii  l  li  l 

for  the  permissible  values  of  (u^.u  )  .  The  log  likelihood  is  then 
n 

log  L  *  7  [w  (log  R  +  log  ui)  +  u  w,  log  (P.R.  +  CwR  ) 

i=p  1  1  ii  ii  i 

+  u.w.  log(P.R.  +  CwR.)]  .  (10) 

Equations  for  maximum  likelihood  estimation  can  be  written  down  from  (10) 


The  following  is  suggested  as  a  possible,  plausible  implementation 
of  this  model: 

1.  The  parameter  »  varies  across  examinees  but  not  across 
it  ems . 

T.  E  P|(  ")  is  a  three-parameter  logistic  or  normal  ogive 
function  of  with  positive  slope. 

3.  R.  E  R.('  )  has  the  same  general  mathematical  form  as  P.(-  ) 

but  different  parameters;  in  particular,  the  slope  is  negative, 
the  lower  asymptote  is  zero. 

The  foregoing  will  be  assumed  in  the  further  discussion  of  this  model. 

There  are  five  item  parameters  in  the  plausible  implementation 
(three  for  P  ,  two  for  R  )  and  two  examinee  parameters  (  *  and  ••  ). 
It  may  not  always  be  practical  to  estimate  this  many  item  parameters.  In 
any  case,  the  model  is  an  aid  to  clear  thinking  about  the  proper  handling 
of  omitted  responses. 

Some  special  cases  of  the  model  are  considered  in  the  remaining 
sections.  The  purpose  is  to  gain  further  insight  into  the  implications 
of  the  model.  These  special  cases  are  not  recommended  for  practical 
use. 

4.  Special  Case:  No  Omitting 
When  w  =  0  ,  we  must  have  w^  =  0  ,  and  (9)  becomes 

n  u,  1— u . 

f(ulf...,un)  =  n  p11(i  -  p1>  1  (ii) 

i  =  l 


where 
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p.  £  P.R.  +  CR.  .  (12) 

i  l  i  l 

Since  (11)  has  Che  same  form  as  (1),  we  have  here  the  usual  item  response 
theory  model  for  dichotomous  items,  omits  being  barred,  except  that 
now  the  item  response  function  has  the  special  form  given  by  (12). 

This  item  response  function  p^(")  need  not  be  a  monotonic 
increasing  function  of  ■■  .  The  lower  asymptote  of  p^(-  )  is  C.  ; 
if  c^  ,  the  lower  asymptote  of  P  (  )  ,  is  less  than  C  ,  the 
probability  of  success  may  decrease  at  first  as  increases,  before 

finally  increasing  to  the  upper  asymptote  at  1.  This  is  a  desirable 
feature:  Examinees  with  sufficiently  low  •  can  only  guess  randomly, 

examinees  with  higher  may  be  misinformed  and  may  do  less  well  than 

a  random  guess.  Model  (11)  -  (12)  was  suggested  bv  Samej ima  (1979). 

5.  Special  Case:  Equivalent  Items,  Knowledge  or  Random  Guessing 

Under  the  knowledge-or-random-guessing  assumption,  the  examinee 
either  knows  the  answer  to  a  particular  item  or  guesses  at  random  or 
omits  it.  For  purposes  of  the  present  model,  the  assumption  is 
represented  by  the  case  where  P^(  )  =  1  for  all  e  . 

When  the  test  is  composed  of  equivalent  items,  the  subscript  i 
can  be  dropped  from  functions  of  parameters.  Denote  the  number  of 
omitted  items  by  w  5  X  w  *  the  number  of  wrong  answers  by 

W  =  ■  .  u  .  w .  . 

til 

1 


i 
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The  log  likelihood  (10)  is  now 

log  L  =  w  log  R  +  w  log  u  +  x  log(R  +  C-^R) 

+  W(log  C  +  log  u>  +  log  R)  .  (13) 

If  we  differentiate  this  with  respect  to  ^  and  set  the  result  eaual 
to  zero,  we  obtain  the  likelihood  equation 

-  -  — M  .  o  .  (14) 

u  R  +  C^R  7 

where  -  and  R  denote  maximum  likelihood  estimators. 

The  likelihood  equation  for  f  is  seen  to  be 


i  log  L  _  1  log  L  JR 

=  ~,R  3  s  "  u 

or  simply  j  log  L/^R  =  0  .  It  is  convenient  for  some  purposes  to 
think  of  R  itself  as  the  ability  parameter,  since  R  =  R  (')  is  a 
one-to-one  monotonic  transformation  of  the  parameter  {•  . 

The  remaining  likelihood  equation  is  thus  seen  to  be 


w  x(— 1  +  Cui)  +  W 
R  R  +  CwR  R 


(15) 


The  maximum  likelihood  estimators  of  <*>  and  of  0  (or  R  )  are  the 


roots  of  (14)  and  (15). 


* 
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Rewrite  (14)  and  (15): 


w  W  =  Cx 

to  Cu)  +  R/R 


(16) 


w  +  W 


x ( 1  —  Cw) 
Cw  +  R/R 


(17) 


Eliminating  R  from  these  two  equations,  we  have 


U! 


Clearing  fractions,  we  find  the  maximum  likelihood  estimator 


w 

~  w  +  W/(l  -  C) 


(18) 


Solve  (17)  to  obtain 


R 

R 


x(l  -  Coj) 
w  +  Vi 


Cw 


1. 

R 


Using  (18), 
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) 


From  (18)  and  (19), 


(19) 


nR 

a  very  reasonable  result.  It  says  that  the  estimated  proportion  of 
omits  equals  the  actual  number  of  omits  divided  by  the  estimated 
number  of  items  on  which  the  examinee  is  totally  ignorant.  Similarly, 
from  (19), 


nR 


(21) 


where  A  E  1/C  .  This  shows  that  the  estimated  number  of  items 
known  by  the  examinee  is  given  by  the  usual  'correction  for  guessing' 


formula. 
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1  Dr.  Norman  J.  Kerr 

Chief  of  Naval  Technical  Training 
Naval  Air  Station  Memphis  (75) 
Millington,  TN  38054 

1  Dr.  William  L.  Maloy 

Principal  Civilian  Advisor  for 
Education  and  Training 
Naval  Training  Command,  Code  00A 
Pensacola,  FL  32508 

1  CAPT  Richard  L.  Martin,  USN 
Prospective  Commanding  Officer 
USS  Carl  Vinson  (CNV-70) 

Newport  News  Shipbuilding  and 
Drvdock  Co. 

Newport  News,  VA  23607 

1  Dr.  Janes  McBride 

Navy  Personnel  R  6  D  Center 
San  Diego,  CA  92152 

1  Mr.  William  Nordbrock 

Instructional  Program  Development 

Building  90 

NET-PDCD 

Great  Lakes  NTC ,  IL  60088 

I  Library,  Code  P201L 

Navy  Personnel  R  &  D  Center 
San  Diego,  CA  92152 
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6  Commanding  Officer 

Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

1  Psychologist 

ONR  Branch  Office 
Building  114,  Section  D 
666  Summer  Street 
Boston,  MA  02210 

1  Office  of  Naval  Research 
Code  437 

800  North  Quincy  Street 
Arlington,  VA  22217 

5  Personnel  and  Training  Research 
Programs 
Code  458 

Office  of  Naval  Research 
Arlington,  VA  22217 

1  Psychologist 

ONR  Branch  Office 
536  S.  Clark  Street 
Chicago,  IL  60605 

I  Psychologist 

ONR  Branch  Office 
1030  East  Green  Street 
Pasadena,  CA  91101 

1  Office  of  the  Chief  of  Naval  Operations 
Research  Development  and  Studies  Branch 
OP-115 

Washington,  DC  20350 

1  The  Principal  Deputy  Assistant 
Secretary  of  the  Navy  (MRA&L) 

4E780,  The  Pentagon 
Washington,  DC  22203 


1  Director,  Research  and 
Analysis  Division 
Plans  and  Policy  Department 
Navy  Recruiting  Command 
4015  Wilson  Boulevard 
Arlington,  VA  22203 

1  Mr.  Arnold  Rubenstein 

Office  of  Naval  Technology 
800  N.  Quincy  Street 
Arlington,  VA  22217 

1  Dr.  Worth  Scanland,  Director 
Research,  Development,  Test 
and  Evaluation 
N-5 

Naval  Education  and  Training  Command 
NAS 

Pensacola,  FL  32508 

1  Dr.  Robert  G.  Smith 

Office  of  Chief  of  Naval  Operations 
OP-987H 

Washington,  DC  20350 

1  Dr.  Alfred  F.  Smode 

Training  Analysis  and  Evaluation  Group 
Department  of  the  Navy 
Orlando,  FL  32813 

1  Dr.  Richard  Sorensen 

Navy  Personnel  R  &  D  Center 
San  Diego,  CA  92152 

1  Mr.  J.  B.  Sympson 

Naval  Personnel  R  &  D  Center 
San  Diego,  CA  92152 

1  Dr.  Ronald  Weitzman 
Code  54  WZ 

Department  of  Administrative  Services 
U.S.  Naval  Postgraduate  School 
Monterey,  CA  93940 
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1  Dr.  Robert  Wisher 
Code  309 

Navy  Personnel  R  &  D  Center 
San  Diego,  CA  92152 

1  Dr.  Martin  F.  Wiskoff 

Navy  Personnel  R  &  D  Center 
San  Diego,  CA  92152 

1  Mr.  John  H.  Wolfe 
Code  P310 

U.S.  Navy  Personnel  Research 
and  Development  Center 
San  Diego,  CA  9/152 

1  Mr.  Ted  M.  1.  Yellen 

Technical  Information  Office 
Code  201 

Navy  Personnel  R  &  D  Center 
San  Diego,  CA  92152 

Array 

1  Technical  Director 

U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Myron  Fischl 

U.S.  Array  Research  Institute  for  the 
Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Michael  Kaplan 

U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 


1  Dr.  Milton  S.  Katz 
Training  Technical  Area 
U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

2  '.’r.  Harold  F.  O'Neil,  Jr. 

Attn:  PERI -OK 

Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  LTC  Michael  Plummer 

Chief,  Leadership  and  Organizational 
Effectiveness  Division 
Office  of  the  Deputy  Chief  of  Staff 
for  Personnel 
Department  of  the  Army 
The  Pentagon 
Washington,  DC  20301 

I  Dr.  James  L.  Raney 

U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Mr.  Robert  Ross 

U.S.  Army  Research  Institute  for 
the  Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Robert  Sasnor 

U.S.  Array  Research  Institute  for 
the  Social  and  Behavioral  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Commandant 

U.S.  Army  Institute  of  Administration 

Attn:  Dr.  Sherrill 

Ft.  Benjamin  Harrison,  IN  4625H 

1  Dr.  Joseph  Ward 

U.S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 
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Air  Force 


1  Air  Force  Human  Resources  Laboratory 
AFHRL/MPD 

Brooks  Air  Force  Base,  TX  78235 

1  U.S.  Air  Force  Office  of 
Scientific  Research 
Life  Sciences  Directorate 
Bolling  Air  Force  Base 
Washington,  DC  20332 

1  Dr.  Earl  A.  Allulsi 
HQ,  AFHRL  (AFSC) 

Brooks  Air  Force  Base,  TX  78235 

1  Dr.  Genevieve  Haddad 
Program  Manager 
Life  Sciences  Directorate 
AFOSR 

Bolling  Air  Force  Base 
Washington,  DC  20332 

1  Dr.  David  R.  Hunter 
AFHRL/MOAM 

Brooks  Air  Force  Base,  TX  78235 

1  Research  and  Measurement  Division 
Research  Branch,  AFMPC/MPCYPR 
Randolph  Air  Force  Base,  TX  78148 

1  Dr.  Malcolm  Ree 
AFHRL/MP 

Brooks  Air  Force  Base,  TX  78235 


Marines 


1  Dr.  H.  William  Greenup 
Education  Advisor  (E031) 
Education  Center,  MCDEC 
Quantico,  VA  22134 


1  Director,  Office  of  Manpower 
Utilization 

HQ,  Marine  Corps  (MPU) 

BCB,  Building  2009 
Quantico,  VA  22134 

1  Special  Assistant  for  Marine 
Corps  Matters 
Code  100M 

Office  of  Naval  Research 
800  N.  Quincy  Street 
Arlington,  VA  22217 

1  MAJ  Michael  L.  Patrow,  USMC 
Headquarters,  Marine  Corps 
Code  MPI-20 
Washington,  DC  20380 

1  Dr.  A.  L.  Slafkosky 
Scientific  Advisor 
Code  RD-1 

HQ,  U.S.  Marine  Corps 
Washington,  DC  20380 


Coast  Guard 


1  Chief,  Psychological  Research  Branch 
U.S.  Coast  Guard  (G-P-1/2/TP42) 
Washington,  DC  20593 

1  Mr.  Thomas  A.  Warm 

U.S.  Coast  Guard  Institute 
P.0.  Substation  18 
Oklahoma  City,  OK  73169 


Other  DoD 


1  DARPA 

1400  Wilson  Boulevard 
Arlington,  VA  22209 
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12  Defense  Technical  Information  Center 
Cameron  Station,  Building  5 
Attn:  TC 

Alexandria,  VA  22314 

1  Dr.  William  Graham 
Testing  Directorate 
MEPCOM/MEPCT-P 
Ft.  Sheridan,  IL  60037 

1  Director,  Research  and  Data 
OASD  (MRA&L) 

3B919,  The  Pentagon 
Washington,  DC  20301 

1  Military  Assistant  for  Training 
and  Personnel  Technology 
Office  of  the  Under  Secretary  of 

Defense  for  Research  and  Engineering 
Room  3D129,  The  Pentagon 
Washington,  DC  20301 

1  Dr.  Wayne  Sellman 

Office  of  the  Assistant  Secretary 
of  Defense  (MRA&L) 

2B269  The  Pentagon 
Washington,  DC  20301 


Civil  Government 


1  Mr.  Richard  McKillip 
Personnel  R  &  D  Center 
Office  of  Personnel  Management 
1900  E  Street,  NW 
Washington,  DC  20415 

1  Dr.  Andrew  R.  Molnar 

Science  Education  Development 
and  Research 

National  Science  Foundation 
Washington,  DC  20550 


1  Dr.  H.  Wallace  Sinaiko 
Program  Director 
Manpower  Research  and 
Advisory  Services 
Smithsonian  Institution 
80'  North  Pitt  Street 
Alexandria,  VA  22314 

1  Dr.  Vern  W.  Urry 

Personnel  R  &  D  Center 
Office  of  Personnel  Management 
1900  E  Street,  NW 
VJashington,  DC  20415 

1  Dr.  Joseph  L.  Young,  Director 
Memory  and  Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 


Non-Government 


1  Dr.  James  Algina 

University  of  Florida 
Gainesville,  FL  32611 

1  Dr.  Erling  B.  Andersen 
Department  of  Statistics 
Studiestraede  6 
1455  Copenhagen 
DENMARK 

1  Psychological  Research  Unit 

Department  of  Defense  (Army  Office) 
Campbell  Park  Offices 
Canberra,  ACT  2600 
AUSTRALIA 

1  Dr.  Isaac  Be  jar 

Educational  Testing  Service 
Princeton,  NJ  08541 
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1  CAPT  J.  Jean  Belanger 

Training  Development  Division 
Canadian  Forces  Training  System 
CFTSHQ,  CFB  Trenton 
Astra,  Ontario  KOK  1B0 
CANADA 

1  Dr,  Menucha  Birenbaum 
School  of  Education 
Tel  Aviv  University 
Tel  Aviv,  Ramat  Aviv  69978 
ISRAEL 

1  Dr.  Werner  Birke 

DezWPs  im  Streitkraef teamt 
Postfach  20  50  3 
D-5300  Bonn  2 
WEST  GERMANY 

1  Dr.  R.  Darrell  Bock 

Department  of  Education 
University  of  Chicago 
Chicago,  IL  60637 

1  Liaison  Scientists 

Office  of  Naval  Research 
Branch  Office,  London 
Box  39 

FPO,  NY  09510 

1  Dr.  Robert  Brennan 

American  College  Testing  Programs 

P.0.  Box  168 

Iowa  City,  IA  52240 

1  Dr.  C.  Victor  Bunderson 
WICAT  Inc. 

University  Plaza,  Suite  10 
1160  S.  State  Street 
Orem,  UT  84057 

1  Dr.  John  B.  Carroll 

Psychometric  Laboratory 
University  of  North  Carolina 
Davie  Hall  013A 
Chapel  Hill,  NC  27514 


1  Charles  Myers  Library 
Livingstone  House 
Livingstone  Road 
Stratford 
London  El 5  2LJ 
ENGLAND 

1  Dr.  Kenneth  E.  Clark 

College  of  Arts  and  Sciences 
University  of  Rochester 
River  Compus  Station 
Rochester,  NY  14627 

1  Dr.  Norman  Cliff 

Department  of  Psychology 
University  of  Southern  California 
University  Park 
Los  Angeles,  CA  90007 

1  Dr.  William  E.  Coffman 

Director,  Iowa  Testing  Programs 
334  Lindquist  Center 
University  of  Iowa 
Iowa  City,  IA  52242 

1  Dr.  Meredith  P.  Crawford 

American  Psychological  Association 
1200  17th  Street,  N 
Washington,  DC  20036 

1  Dr.  Fritz  Drasgow 

Yale  School  of  Organization  and 
Management 
Yale  University 
Box  IA 

New  Haven,  CT  06520 

1  Dr.  Mike  Durmeyer 

Instructional  Program  Development 

Building  90 

NET-PDCD 

Great  Lakes  NTC,  IL  60088 

1  ERIC  Facility-Acquisitions 
4833  Rugby  Avenue 
Bethesda,  MD  20014 
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1  Dr.  A.  J.  Eschenbrenner 
Dept.  E422,  Bldg.  81 
McDonald  Douglas  Astronautics  Co. 
P.0.  Box  516 
St.  Louis,  MO  63166 

1  Dr.  Benjamin  A.  Fairbank,  Jr. 

Me Fa nn -Gray  and  Associates,  Inc. 
5825  Callaghan 
Suite  225 

San  Antonio,  TX  78228 

1  Dr.  Leonard  Feldt 

Lindquist  Center  for  Measurement 
University  of  Iowa 
Iowa  City,  IA  52242 


1  Dr.  Richard  L.  Ferguson 

The  American  College  Testing  Program 

P.0.  Box  168 

Iowa  City,  IA  52240 

1  Dr.  Victor  Fields 

Department  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1  Unlv.  Prof.  Dr.  Gerhard  Fischer 
Psychologi sches  Institut  der 
Universitat  Wien 
Liebiggasse  5/3 
A  1010  Wien 
AUSTRIA 

1  Prof.  Donald  Fitzgerald 
University  of  New  England 
Armidale,  New  South  Wales  2351 
AUSTRALIA 

1  Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organization 
Suite  900 

4330  East  West  Highway 
Washington,  DC  20014 


1  Dr.  John  R.  Frederiksen 
Bolt,  Beranek,  and  Newman 
50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Robert  Glaser 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1  Dr.  Daniel  Gopher 

Industrial  and  Management  Engineering 
Technion-Israel  Institute  of 
Technology 
Haifa 
ISRAEL 

Dr.  Bert  Creen 
Department  of  Psychology 
Johns  Hopkins  University 
Charles  and  34th  Streets 
Baltimore,  MD  21218 

Dr.  Ron  Hambleton 
School  of  Education 
University  of  Massachusetts 
Amherst,  MA  01002 

Dr.  Delwyn  Harnisch 
University  of  Illinois 
242b  Education 
Urbana,  IL  61801 

Dr.  Chester  Harris 
School  of  Education 
University  of  California 
Santa  Barbara,  CA  93106 

Dr.  Lloyd  Humphreys 
Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 
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1  Library 

HumRRO/We stern  Division 
27857  Berwick  Drive 
Carmel,  CA  93921 

1  Dr.  Steven  Hunka 

Department  of  Education 
University  of  Alberta 
Edmonton,  Alberta 
CANADA 

1  Dr.  Jack  Hunter 

2122  Coolidge  Street 
Lansing,  MI  48906 

1  Dr.  Huynh  Huynh 

College  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

1  Prof.  John  A.  Keats 

Department  of  Psychology 
University  of  Newcastle 
Newcastle,  New  South  Wales  2308 
AUSTRALIA 

1  Mr.  Jeff  Kelety 

Department  of  Instructional  Technology 
University  of  Southern  California 
Los  Angeles,  CA  90007 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Building 
University  of  Illinois 
Champaign,  IL  61801 

1  Dr.  Charles  Lewis 

Faculteit  Sociale  Wetenschappen 
Ri jksuniversiteit  Groningen 
Oude  Boteringestraat  23 
9712GC  Groningen 
NETHERLANDS 


1  Dr.  Robert  Linn 

College  of  Education 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  James  Lumsden 

Department  of  Psychology 
University  of  Western  Australia 
Nedlands,  Western  Australia  6009 
AUSTRALIA 

1  Dr.  Gary  Marco 

Educational  Testing  Service 
Princeton,  NJ  08541 

1  Dr.  Scott  Maxwell 

Department  of  Psychology 
University  of  Houston 
Houston,  TX  77004 

1  Dr.  Samuel  T.  Mayo 

Loyola  University  of  Chicago 

820  North  Michigan  Avenue 

Chicago,  IL  6061 1  v, 

1  Prof.  Jason  Millman 

Department  of  Education 
Stone  Hall 
Cornell  University 
Ithaca,  NY  14853 

1  Dr.  Melvin  R.  Novi ck 

356  Lindquist  Center  for  Measurement 
University  of  Iowa 
Iowa  City,  IA  52242 

1  Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 
400  Army  Navy  Drive 
Arlington,  VA  22202 

1  Dr.  Wayne  M.  Patience 

American  Council  on  Education 
GED  Testing  Service,  Suite  20 
One  Dupont  Circle,  NW 
Washington,  DC  20036 
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1  Dr.  James  A.  Paulson 

Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

1  Mr.  Luigi  Petrullo 

2 A 3 1  North  Edgewood  Street 
Arlington,  VA  22207 

1  Dr.  Diane  M.  Ransey-Klee 

R-K  Research  and  System  Design 
3997  Ridgemont  Drive 
Malibu,  CA  90265 

1  Mr.  Minrat  M.  L.  Rauch 
P  II  4 

Bundesninisterium  der  Verteidigung 
Postfach  1328 
D-53  Bonn  1 
GERMANY 

1  Dr.  Mark  D.  Reckase 

Educational  Psychology  Department 
University  of  Mi ssouri-Columbia 
4  Hill  Ha’ll 
Columbia,  MO  65211 

1  Dr.  Andrew  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.,  NW 
Washington,  DC  20007 

1  Dr.  Leonard  L.  Rosenbaum,  Chairman 
Department  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1  Dr.  Ernst  Z.  Rothkopf 
Bell  Laboratories 
600  Mountain  Avenue 
Murray  Hill,  NJ  07974 

1  Dr.  Lawrence  Rudner 
403  Elm  Avenue 
Takoma  Park,  MD  20012 


1  Dr.  J.  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia,  SC  29208 

1  Prof.  Fumiko  Sanejima 
Department  of  Psychology 
University  of  Tennessee 
Knoxville,  TN  37916 

1  Dr.  Kazuo  Shigemasu 
University  of  Tohoku 
Department  of  Educational  Psychology 
Kawauchi,  Sendai  980 
JAPAN 

1  Dr.  Edwin  Shirkey 

Department  of  Psychology 
University  of  Central  Florida 
Orlando,  FL  32816 

1  Dr.  Robert  Smith 

Department  of  Computer  Science 

Rutgers  University 

New  Brunswick,  NJ  08903 

1  Dr.  Richard  Snow- 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

1  Dr.  Robert  Sternberg 
Department  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 

1  Dr.  Patrick  Suppes 

Institute  for  Mathematical  Studies  in 
the  Social  Sciences 
Stanford  University 
Stanford,  CA  94305 


1  Dr.  Hariharan  Swaminathan 

Laboratory  of  Psychometric  and 
Evaluation  Research 
School  of  Education 
University  of  Massacuusett s 
Amherst,  MA  01003 

1  Dr.  Kikumi  Tatsuoka 

Computer  Based  Education  Research 
Laboratory 

252  Engineering  Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  David  Thissen 

Department  of  Psychology 
University  of  Kansas 
Lawrence,  KS  66044 

1  Dr.  Robert  Tsutakawa 

Department  of  Statistics 
University  of  Missouri 
Columbia,  MO  65201 

1  Dr.  Howard  Wainer 

Educational  Testing  Service 
Princeton,  NJ  08541 

1  Dr.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  East  River  Road 
Minneapolis,  MN  55455 

1  Dr.  Susan  E.  Whitely 
Psychology  Department 
University  of  Kansas 
Lawrence,  KS  66044 

1  Dr.  Wolfgang  Wildgrube 
Streitkraef teamt 
Box  20  50  03 
D-5300  Bonn  2 
WEST  GERMANY 


